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Introduction 


An  Increasing  use  is  being  made  of  non -parametric  tests,  i.e., 
tests  of  significance  which  are  not  based  on  the  usual  assumption  that  all 


distributions  are  normal.  Many  psychologists  have  assumed  that  when  non 


normality  or  heterogenous  variance  la  encountered  in  the  data,  then  analysis 


I of  variance  by  ranks,  the  Mann-Whitney  U Test  (1947),  Festlnger's  d (1946) 


or  some  similar  non-parametrlc  technique  is  appropriate 


The  purpose  of  this  paper  is  to  point  out  that  non-parametric  tests 


also  have  basic  assumptions  which  must  be  met,  and  that  often  the  factors 


which  bar  the  use  of  the  parametric  tests  also  violate  the  assumptions 


underlying  most  of  the  non-parametrlc  tests.  I am  indebted  to  Professor 


Leon  Festlnger  for  calling  this  to  my  attention  (personal  communication). 

Almost  all  non-parametric  tests  (in  particular,  those  which  use  a rank 
order  procedure)  assume  that  all  distributions  are  identical.  This  means 


that  if  the  variances  are  homogeneous,  or  third  moments  are  not  equal,  etc 
these  non-parametrlc  tests  are  Inappropriate  for  testing  the  equality  of 


To  fix  our  ideas,  the  following  three  non-parametric  tests  will  be 


considered 


(1)  The  Kruskal-Wallin  H Test  (one-way  analysis  of  variance  by  ranks) 
(1952) 


(2)  The  Kendall  V Test  for  differences  in  correlated  means.  (1948) 


(3)  The  test  for  difference  in  medians.  (Westenberg,  1948) 


What  are  the  assumptions  underlying  these  procedures?  When  can  they 
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Appropriately  be  substituted  for  the  usual  parametric  tests? 

Tbs  Kruahal-Wallle  H Test 

Let  us  suppose  that  we  have  dram  c samples  from  the  population 


and  we  wish  to  test  the  hypothesis  that  all  c means  are  equal.  The 


first  step  is  to  rank  all  N observations  without  regard  to  the  sample 


from  which  they  come.  Then  if  there  are  no  ties,  compute 


3(N+i) 


N(N+1)  1-1  n 


where  c » the  number  of  samples 

n^  - the  number  of  observations  in  the  i*h  sample 
N - the  total  number  of  observations 


Kruskal  and  Hal 11a  (1952,  page  586)  state  that  "If  the  samples 
come  from  Identical  continuous  populations  and  the  n^  are  not  too 
small..."  H is  distributed  as  chi-square  with  c-1  degrees  of  freedom 
He  may  disregard  the  assumption  of  a large  nt,  for  when  is  too 
small,  it  simply  means  that  special  distribution  tables  must  be  con- 


sulted. H is  still  a valid  measure  of  the  discrepancy  between 


Later  they  state  that  "if  the  data  are  matched  H is  not  appropriate 
and  Friedman's  chi-square  test  (or  equivalently  Kendall's  H)  should 


There  then,  are  the  two  fundamental  assumptions  underlying  the  use 


(1)  The  samples  come  from  Identical  continuous  populaob 


(2)  The  observations  are  Independent  of  one  another 
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The  first  assumption  wans  that  tha  samples  aay  ba  drawn  ftoa 
populations  with  a non-no rwl  distribution,  but  all  samples  must  ba 
drawn  fron  populations  with  tha  saw  non-norwl  distribution.  For 
example,  if  ws  haws  drawn  ona  ssapls  fron  a population  whosa  distribution 
has  a positive  sksw  and  a sscond  sanpls  fron  a population  whosa 
distribution  has  a negative  sksw,  tha  H tsst  cannot  ba  appliad  to  saa 
if  tha  two  sanpla  wans  diffar  significantly. 

Tha  sacond  assuaptlon  is  tha  usual  randon  stapling  raqulranant  of 
statistical  independence . In  particular,  observations  should  ba  drawn 
so  that  no  systenatlc  way  of  wtchlng  tha  observations  will  produce  s 
correlation  (linear  or  non-linear)  between  the  Hatched  observations.  For 
exaaple,  if  we  have  obtained  treatwnt  scores  A sad  B fron  the  saw  indivi- 
dual, the  H test  cannot  be  used  to  test  the  difference  between  the  wans 
of  Treatwnt  A and  Treatwnt  B.  Obviously,  this  independence  assumption 
cannot  be  satisfied  if  the  nuaber  of  test  scorss  exceeds  the  number  of 
subjects.  When  there  are  two  or  more  scores  per  individual  the  scores 
will,  in  general,  be  dependent  upon  ow  another,  in  a probability  sense. 

This,  however,  is  the  saw  as  the  usual  analysis-of -variance  assuaptlon 
of  Independence  of  observations.  Hsre  too,  the  samples  oust  be  Independently 
drawn.  The  use  of  three  or  aore  per  subject  will,  in  general,  force  the 
use  of  a auch  aore  coaplicated  technique— the  aultl-varlate  analysis  of 
verlance  (Rao,  1951,  pp.  239-246;  C.  B.  P.  Box,  1954,  pp.  484-498). 

It  has  been  custowry  for  sow  workers  to  resort  to  non-parawtrlc 
tests  when  the  varlences  are  heterogeneous.  Obviously,  the  "identical 
distribution1'  assuaptlon  bars  the  use  of  the  H test  in  this  case.  If 
we  allow  two  staples  to  have  very  different  varlences,  it  is  easy  to 
construct  an  example  where  H is  significant,  even  when  the  two  wens  are 
identical! 


Xa  Table  l consider  the  syneetrlc  distribution  of  Treatment  A and 
tha  skewed  distribution  of  Treatment  B.  Bara  tha  H tast  la  significant 
at  tha  l parcant  level  although  tha  aaana  ara  equal . 

Tabla  1 about  hors 

It  nust  not  ba  thought  that  tha  H taat  la  simply  inaceurata  In  this 
caaa  and  that  sons  othar  non-par  enetric  tast  would  not  give  a significant 
dlffaranca.  On  tha  contrary  tha  H tast  givas  essentially  tha  sans  rasults 
as  any  othar  taat  basad  on  rank -order,  runs,  randomisation,  ate. 

Tha  reason  for  this  Is  that  all  non-par anatrlc  rank-ordar  tests  seen 
to  ba  basad  on  assumption  1,  that  sanplas  cons  fron  identical  continuous 
populations.  Thera  do  not  seen  to  ba  any  rank-order  tests  basad  on  tha 
notion  that  tha  sanplas  cons  fron  several  different  populations  (with 
respect  to  variance,  skewness,  ate.),  where  the  population  naans  ara  aqual. 

Significance  tests  do  not  always  have  to  conform  to  their  assumptions 
to  ba  useful.  Per  example,  tha  t tast  assumes  that  tha  two  samples  ara 
drawn  fron  distributions  with  a comnon  variance.  Tat  Welch  (1938)  waa 
able  to  show  that  as  long  as  tha  two  sanplas  ware  of  equal  sisa.  one 
variance  could  ba  tan  times  tha  othar  without  seriously  disturbing  tha  3 
parcant  level  of  t.  The  term  "robust"  has  bean  used  by  Box  (1954)  to 
denote  procedures  Ilka  tha  t tast  which  ara  maximally  sensitive  to  differences 
In  tha  crucial  parsentars  being  examined  (such  as  tha  sample  averages)  but 
ara  fairly  Insensitive  to  differences  In  some  of  tha  Irrelevant  parameters 
(such  as  varlaneas,  fourth  monsnts,  ate.) 

Bow  "robust"  la  the  B tast?  Under  what  conditions  can  tha  assumption 
of  Identical  population  distributions  be  ignored?  Kruskal  and  Wallis 
conjecture  that  If  two  populations  have  symmetric  distributions,  than 


dtfferancaa  to  *erl ability  will  uot  affect  tha  significance  level*  of  the 
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H teat  wry  auch,  (1952,  pp.  S99>600). 

11m  multi  of  our  inquiry  into  the  H tost  can  bo  suamarised  os 
follows: 

(1)  A significant  value  of  H indlcatos  that  ths  populations  differ, 
but  tho  slgnlflcanco  nay  be  duo  to  differences  in  variances, 
third  nowents,  etc.,  and  not  necessarily  to  differences  in 

the  weans. 

(2)  The  H test  for  difference  in  weans  is  probably  robust  when 
population  distributions  are  sywastrlc. 

Kendall* s W Test  for  Differences  in  Correlated  Means 

It  has  already  been  wsntloned  that  Kendall's  W is  appropriate  when 
the  data  are  watched.  Suppose  that  n treatments  have  been  applied  to 
each  of  a subjects.  A score  matrix  would  result  with  m rows  and  n columns. 

A row  would  represent  the  n treatment  scores  of  one  subject.  These 
a treatment  scores  can  be  ranked.  There  will  be  m such  rankings.  If 
tho  treatment  that  is  highest  for  one  subject  is  highest  for  all  subjects 
there  will  tend  to  be  a significantly  positive  rank-order  correlation 
between  any  two  subjects. 

In  gsneral,  if  one  treatment  tends  to  be  consistently  higher  then 
another,  the  average  rank  order  correlation  among  all  m subjects  will 
be  significant.  Kendall  and  Smith  (1939)  proposed  a statistic,  W,  which 
is  a monotonlc  function  of  the  average  Spearman  rank-order  coefficient. 

If  equals  one  if  the  n rankings  agree  perfectly,  and  is  saro  or  near-sero 
when  the  average  rank-order  coefficient  is  as  low  as  possible. 

Friedman  (1937)  had  previously  suggested  a statistic, Xf»  roughly 
distributed  as  chi-square  for  large  samples,  which  is  also  a monotonlc 
function  of  the  average  rank-order  correlation.  Since  the  rank  order 
of  the  n treatments  for  any  subject  is  Independent  of  the  correlations 
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lutifiin  the  treatment  moth,  eh*  PtMhb  ehi««quari  at  at  la  tie  (or  tha 
filial 1 W)  can  ba  used  Khan  tha  treatment  means  ara  corralatad. 

Tha  fundamental  aaauaptlona  of  tha  Prladaan  chl-sqamre  taat  ara: 

(1)  Tha  a traataant  aaaplaa  coaa  froa  idantlcal  populatlona 

(2)  Tha  a rows  of  obaarvatlona  ara  Independent  of  ona  anothar . 
Essentially  thaaa  ara  tha  aaaa  raatrlctlona  as  thoaa  of  tha  B taat, 

and  aaaa  to  load  to  alallar  dlfflcultlaa.  for  Instance,  If  tha  c traataanta 
produea  unaqual  variances,  a significant  chi-square  can  ha  obtalnad  avan 
ahan  tha  a aaana  ara  Idantlcal.  If  Tabla  1 la  analysed  as  a Prladaan  9 
by  2 scorn  aatrlx  (with  two  acorns  par  Individual)  than 


Al 


(»!  ■!<.<■  »J 

re  (c+1) 


2 


5.444  with  (c-l)»l  da  gran  of 
fraadoa. 


(2) 


This  Prladaan  chi-square  la  significant  at  tha  .02  laval.  (Tha  chi -square 
distribution  is  generally  not  an  accurats  approximation  to  tha  distribution 
of  Prladaan' a chi-square  for  c • 2 and  R.  A.  Plsher's  exact  binominal  sign- 
teat  should  be  used.  In  this  case  the  sign  test  rejects  the  null  hypothesis 
of  identical  means  at  the  .04  level.) 

Mo  Investigations  have  been  aade  of  the  robustness  of  Prladaan' a test. 
It  aaaaa  likely  that  It  la  robust  under  tha  same  conditions  as  tha  H test, 
l.e.,  when  the  n traataant  distributions  are  symmetrical. 

Tha  Taat  for  Differences  In  Medians 

Tha  mathematical  statistician  sonatinas  follows  tha  adage  "When  you 
can't  ha  near  tha  girl  you  love,  then  you  love  the  girl  you're  near." 

If  tha  question  about  differences  between  aaaaa  can't  be  answered 
satisfactorily,  let's  ask  sons  other  related  question  which  cm  be 
answered.  Vhen  tha  distributions  ara  not  symaatrlc,  tha  aedlan  and  node 
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no  longer  co  Inc  id*  and  It  becomes  a natter  of  preference,  utility,  and 
convenience  aa  to  which  naaaur*  of  central  tandancy  should  ba  usad.  Why 
not  devise  a taat  for  tha  dlffarancaa  in  medians?  After  all,  if  the 
dlatrlbutlona  are  badly  skewed,  aost  statisticians  will  be  using  tha 
nedian,  rather  than  the  naan,  for  interpretative  purposes. 

Westenberg  (1948)  and  Mood  and  Brown  (1950)  have  discussed  such  a 
taat  for  the  differences  in  nedlans . 

The  basic  step  is  to  arrange  tha  observations  from  the  c ssnples  in 
order  of  sice.  Then  the  0*^1 >th  observation  is  chosen  as  the  best 
estinata  of  the  population  nedian.  Now  the  c ssnples  can  ba  conpared  in 
terns  of  tha  numbers  above  tha  estimated  population  nedian.  This  results 
in  a 2 by  c contingency  table.  If  the  c sanplas  differ  markedly  in  the 
proportion  of  cases  above  tha  population  nedian  than  chi-square,  or  the 
more  exact  multinomial  technique,  will  show  tha  significance  of  these 
differences . 

For  ex«pl*  in  Table  1,  the  estimated  population  nedian  is  158.5, 
the  nedian  for  Traatnant  A ia  155,  and  the  nedian  for  Treatment  B is  163. 
Table  2 shows  the  resulting  2 by  c contingency  table.  Both  the  usual 
chi-square  and  Fisher's  exact  2 by  2 test  show  significance  at  tha  .01 

level.  - 

Table  2 about  her* 

The  nedian  tast  can  ba  derived  fron  two  basic  assunptions: 

(1)  Tha  ssnples  cone  fron  continuous  populations  with  a common 

nedian 

(2)  Tha  observations  are  Independent  of  one  another 

It  follows  that  for  tha  1th  sample,  the  number  of  cases  above  the 
population  nedian  is  distributed  as  the  binomial 


.«  • - ..  - v a . 


(1/2+1/2  * ) 
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vhir* 

the  exponent  of  X the  number  above  the  population  median 

in  the  1th  sample 

n^  la  the  number  of  obaervatlons  in  the 

ith  taapie 

One  of  the  chief  difficulties  la  that  the  common  population  median 
is  estimated  from  the  total  distribution  of  the  samples.  It  does  not 
seem  to  be  known  under  what  conditions  such  an  estlmata  will  be  consistent 
and  efficient.  Thla  in  turn  means  that  generally  it  is  not  known  whether 
the  median  la  biased  or  what  its  power  is. 

The  chief  advantage  la  that  the  assumption  of  Identical  population 
distributions  is  dropped.  The  samples  can  be  drawn  from  any  c populations 
and  tested  for  a common  median  provided  only  that  the  obaervatlons  are 
sampled  in  a random  and  Independent  manner,  as  usual. 

The  median  test  has  the  advantage  over  the  H test  that  differences  in 
variability,  skewness,  etc.,  apparently  do  not  invalidate  the  procedure. 
Any  set  of  diversely  shaped  distributions  with  a common  median  will  havs 
the  bitKMsial  distribution  of  equation  (3). 

If  the  term  "distribution-free"  were  reserved  for  those  methods  which 
do  not  assume  the  equality  of  Irrelevant  paraaaters  (such  as  the  variance, 
skewness,  fourth  moment,  etc.)  then  presumably  much  misapprehension  about 
the  nature  of  these  tests  could  be  avoided.  "Distribution- free"  seems  an 
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appropriate  adjective  for  a technique  which  assumes  equality  only  for  those 
crucial  paramatars  being  tested  for  equality. 

Those  methods  which  assume  Identical  but  non-normal  distribution 
could  possibly  beat  be  descrlbad  as  "normal-dlstrlbutlon-free,"  "non- 
normal," or  "identical  population"  testa.  At  any  rate,  it  seems  rather 
inappropriate  to  lable  as  "non-parametrlc,"  testa  which  are  completely 
dependent  upon  the  values  of  the  ron-cruclal  parameters  in  the  assumed 
non-normal  population. 

^ I I I Ilia  a I - - - ■ II  I I » ■ ■ ■ — ■ ■ — , , , - -- 
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Recoamendat ions 

On*  purpose  of  this  not*  was  to  set  out  the  conditions  when  the  non* 
parametric  tests  should  end  should  not  be  applied. 

If  we  are  Interested  in  differences  between  means,  the  t test  and 
7 ratio  are  appropriate  when  the  distribution  within  the  populations  being 
sanpled  are  normal  and  have  homogeneous  variance.  These  tests  are 
"robust,"  l.e.,  they  have  approximately  the  tabled  distributions  even  when 
these  assumptions  are  not  met  exactly. 

Case  I — The  population  distributions  are  normal  with  differing 
variances. 

When  the  distributions  are  normal  and  the  number  in  each  sample  is 
large,  then  heterogeneous  variances  will  not  create  much  disturbance. 

If  the  sample  sizes  are  small  but  equal,  then  heterogeneous  variances  will 
still  make  little  difference,  although  for  accuracy,  a correction  should 
be  made  reducing  the  degrees  of  freedom  (G.  E.  P.  Box,  1954,  p.  300). 

But  if  the  sample  sizes  are  small  and  unequal  then  corrections  have 
to  be  applied  to  the  F ratio  as  well  as  the  degrees  of  freedom.  Here  the 
approximation  as  well  as  being  more  complex,  tends  to  be  less  accurate 
then  in  the  case  of  equal  groups. 

It  is  at  this  point  that  rank-order  procedures  like  the  H test  can  be 
applied.  Although,  strictly  speaking,  the  heterogenous  variances  violate 
the  assumption  of  identical  population  distributions,  the  H test  is 
probably  robust  because  the  distributions  are  symmetric.  Of  course  the 
median  test  could  be  used,  for  here  the  median  coincides  with  the  mean, 
but  it  is  almost  certain  that  the  rank-order  tests  will  be  much  more 
sensitive  than  any  dichotomizing  procedure. 

Case  II— The  population  distributions  are  identical  but  non-normal. 

. The  straightforward,  but  painful,  procedure  in  this  case  would  be 
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to  ascertain  the  nature  of  the  conon  distribution  end  either  devise  an 
exact  test  or  transform  to  the  normal  distribution. 


If  the  population  dlatrlbutlon  Is  synetrlcal,  not  such  Is  gained 
in  the  procesa.  With  large  a ample a.  the  tabled  percentage  points  of  the 
t and  P ratios  remain  good  approximations  except  In  pathological  cases 
such  as  the  Cauchy  distribution.  With  small  sample  sizes  and  symmetrical 
non-normal  population  distributions,  the  H test  would  probably  be  a good 
choice. 

When  the  common  population  distribution  Is  skewed,  fairly  large 
samples  are  needed  before  the  t and  F ratios  become  good  approximations 
to  the  tabled  percentage  points.  The  H test  Is  probably  a safer  test  to 
use  than  the  t or  the  F ratio  for  the  usual  sample  sizes.  The  median  test 
is  not  recommended.  Since  the  distributions  are  Identical,  the  medians 
will  differ  between  groups  in  the  same  direction  as  the  means.  The  more 
sensitive  H test  will  tend  to  pick  up  small  differences  which  the  less 
powerful  median  test  is  unable  to  discriminate. 

Case  III--The  population  distributions  are  non-identical  and  non- 
normal. 

Neither  the  t test  nor  the  H test  Is  strictly  appropriate  here.  If 
the  distributions  are  symmetric,  the  H test  will  probably  be  sensitive 
to  differences  In  the  means.  If  the  distributions  are  not  symstrlc  and 
have  unequal  variances,  the  median  test  can  be  used  If  the  median  is 
felt  to  be  a useful  measure  of  the  central  tendency. 

Case  IV— The  observations  are  dependent. 

Most  often  this  Is  encountered  when  repeated  measurements  are 
taken  for  each  subject.  If  the  n scores  for  the  m subjects  form  a multi- 
variate normal  distribution,  then  multivariate  analysis -of -variance  (Rao, 
1952,  pp.  240-244)  Is  the  moet  appropriate  and  sensitive  test  for  differences 


i 


between  the  n correlated  treatment  means 
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Soma  psychological  statisticians  such  as  Edwards  (1950),  Lindquist 
(1940),  and  McNeaar  (1949)  havt  rscoonsnded  that  tha  a by  n scors  matrix 
should  ba  traatad  by  a two-way  analysis  of  variants,  using  tha  lntaractlon 
as  tha  arror  tarm.  For  tha  "batwasn  treatments"  comparison  this  will  ba 
axactly  corract  whan  n»2,  and  approximately  so  whan  n?  2 if  all  lnter- 
corralations  between  tha  n treatments  are  equal  (Box,  1954,  p.  489).  How- 
ever when  tha  correlations  between  treatments  are  unequal,  it  is  likely 
that  the  F ratio  for  the  treatment  effect  will  be  seriously  biased.  The 


'between-subjects"  F ratio  is  even  more  seriously  affected. 

When  the  n treatment  distributions  are  non-normal  but  identical,  then 


try  swiftly 


Kendall's  H is  appropriate.  The  sensitivity  of  W Inc 


as  n increases  and  somewhat  more  slowly  as  m Increases. 

So  far  as  I know,  no  test  has  been  proposed  for  the  case  when  the 


treatment  distributions  are  not  assumed  to  be  identical 


trie  rank-order  testa  such  as  the  Kruskal-Wallis  H 


test,  Friedman's  test  for  differences  in  corrslated  means,  the  Mann 


Whitney  test,  etc.,  assume  that  samples  are  drawn  from  identical  population 
distributions.  Therefore,  when  heterogeneity  of  variance,  differences  in 
skewness,  etc.,  are  found,  these  non-parametrlc  tests  are  not  theoretically 
appropriate  substitutes  for  the  t test  or  F ratio.  Strictly  speaking,  they 
should  only  be  used  with  identical  non-normal  populations.  However,  if 
the  population  distributions  are  symmetric,  the  H test  is  probably  a 
sensitive  test  of  the  differences  in  sample  means. 


The  median  test  can  be  used  without  the  assumption  of  identical  popu 


latlon  distributions,  and  therefore  can  always  be  applied.  But  the  median 


is  not  always  an  appropriate  measure  of  central  tendency.  At  present. 
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wry  little  la  known  of  the  bias  and  power  of  tha  nedian  teat  whan  samples 
are  drawn  from  a spec If lad  eat  of  non-identical  distributions. 

Zt  la  suggested  that  tha  tern  "dletrlbut ion- free"  be  reserved  for 
those  significance  teats  which  do  not  assusw  that  all  populations  are 
identical. 
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Table  1 

Example  of  a Significant  H When  Means  are  Identical 


Treatment  A 


Treatment  B 


Sum 


Score 

Rank 

Score 

Rank 

151 

2 

160 

11 

152 

3 

161 

12 

153 

4 

163 

13 

154 

5 

163 

14 

155 

6 

164 

15 

156 

7 

165 

16 

157 

8 

166 

17 

158 

9 

167 

18 

159 

10 

87 

1 

1395 

54 

N-18 


V 

H - 

M - 


1.796 

12 

18(19) 


542  + H72 


-3(19)  - 7.737 


- *<2h  . 12.789 
18(19) 


. 7.737  (11.789) 

5.052 

2 (11.789-1.796) 
12.789  (1.796) 


■ 18.055  Significant  at  the  1 percent  level 
- .870 


\ Lubln 


Cases  above  158.5 


Bases  below  158.5 


<!S(8)-l/  l2pi8 

1 — ■ 12 
9<9)9<9) 


Table  2 


Treatment  A Treatment  B 


Significant  at  the  .01  level 


b 


AH S bCN-i AHAKSTRIC  T2STS  JISTRL- 

BUTioH-jaaa? 
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